Skip to content

fix(redis): Bound the wait for a secondary connection - #6590

Open
ldelvoye wants to merge 3 commits into
feat/quotas-redis-shadow-writefrom
ldelvoye/fix/quotas-secondary-connect-timeout
Open

ldelvoye wants to merge 3 commits into
feat/quotas-redis-shadow-writefrom
ldelvoye/fix/quotas-secondary-connect-timeout

Conversation

@ldelvoye

@ldelvoye ldelvoye commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Bounds the dual client's wait for a secondary connection to 100ms, so an exhausted or unreachable secondary pool no longer holds the primary connection.

The secondary is acquired in a spawned task, so an acquisition that times out still finishes and fills the pool instead of being cancelled.

dmajere and others added 3 commits October 29, 2026 08:58
An exhausted or unreachable secondary pool held the primary connection
until it resolved, with no limit by default. The secondary is now
acquired in a spawned task capped at 100ms, so a timed-out acquisition
still finishes and fills the pool for later calls.
@ldelvoye
ldelvoye force-pushed the ldelvoye/fix/quotas-secondary-connect-timeout branch from d7894db to 8148abe Compare October 29, 2026 15:59
@ldelvoye ldelvoye changed the title feat(quotas): Support dual writes for the quotas Redis pool fix(redis): Bound the wait for a secondary connection Oct 29, 2026
@ldelvoye
ldelvoye changed the base branch from master to feat/quotas-redis-shadow-write October 29, 2026 15:59
@ldelvoye
ldelvoye added this pull request to stack #6606 October 29, 2026 15:59
@ldelvoye
ldelvoye marked this pull request as ready for review October 30, 2026 16:53
@ldelvoye
ldelvoye requested a review from a team as a code owner October 30, 2026 16:53
@ldelvoye
ldelvoye requested a review from dmajere October 30, 2026 16:53

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8148abe. Configure here.

Comment thread relay-redis/src/real.rs
"timed out acquiring a secondary redis connection",
);
None
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timeouts logged as Sentry errors

Medium Severity

A secondary acquisition timeout is the expected fallback when the pool is slow or exhausted, but this path logs with relay_log::error!, which Relay reports to Sentry. Under that load this can emit one Sentry event per request.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 8148abe. Configure here.

Comment thread relay-redis/src/real.rs

let primary_connection = primary.get_boxed_connection();
let secondary_connection =
tokio::time::timeout(SECONDARY_CONNECTION_TIMEOUT, secondary_task);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unbounded secondary acquisition tasks

Medium Severity

The spawned secondary get_connection is not given its own deadline. After the caller times out, that task keeps waiting, and the default pool wait_timeout is None, so an exhausted secondary leaves one background waiter per request.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 8148abe. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants